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TNTFJ.l.TGENT REMOTE MULTIMODE SENSE AND DISPLAY 
SYSTEM trmiZP^G HAPnC INFORMATION COM PRESSION 

5 Background of the Invention 

^ The present invention relates generally to a system and method of 

providing haptic (position, orientation, force, & tactile) feedback to the 
human operator or a remote .or virtual environment. 

Devices and systems are well known for nieasuring or monitoring one 
10 or more characteristics of an .anatomical part (see for example U.S. Patent 
Nos. 4,986,280, 3,258,007 and 3,364,929). A human cpniroUed position 
sensing device for use in the field of robotics is also well known (see for 
example, U.S. Patent Hos. 4,328,621, 4,534,694, 4,608,525, and 
4,674,048). 

15 

Force Feedback Svstems 

Systems are also well knovm for providing force feedback to human 
operators. Fdrce-reflective devices exist in a variety of forms, from 
joysticks to arm and hand controllers. These devices .serve as input devices 

20 and haptic interfaces for virtual enviroimient nianipulation, or teleoperatipn. 
A significant anaount of research has been conducted on robotic hands and 
force-reflecting controllers for robotic arm naanipulation. While a few 
controllers were anthropomorphic, most were typically tailored to imitate the 
slave robot's kinematics. The advantage of anthropomorphic controllers is 

25 that they imitate the kinematics of the human hand and arm, tois possessing 
the largest range of human operator (user) motion. Yet these past 
anthropomorphic designs were unable to achieve high quality force feedback 

* 

while retaining minimal actuator size, a requirement for system portability. 
So while many forms of force-feedback devices have been developed, few 
30 completely anthropomorphic systems exist. The systems tended to he 

massive and bulky , and required large centralized computational resources 

/ 
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lightly coupled to the sensory system, further preventing an efficient 
integration of the system to the huinan operator. Finally, the systems were 
,non-mo.dular, requiring extensive work to add additional haptic sense and 
display devkes. 

5 

Tactile Systems 

Tactile displays have been the subject of much research activity in the 
past thirty years. While neuroscientists studied the physiologic^ roots of 
human tactile perception, engineers were interested in tactile display devices 
10 that substituted or augmented other senses such as sight and hearing. Tactile 
displays also demonstrated their usefulness in virmal reality simulations and 
telerobotic operations. 

A system utilizing tactile display devices provides die teleoperator 
with additional information, such as contact locations, diat could not be 
15 adequately displayed by force alone. Tactile displays also simpliiy the 

requirements placed upon force reflectidn. A combination of tactile display 
and force reflection is haptically more effective, technically less challenging 
and economically more feasible than force reflection alone. Therefore, a 
' high performance force reflectivis exoskeleton master should incorporate 
20 tactile displays. 

> Tactile display technologies have slowly evolved in the past thirty 

years. Many different tactile display technologies are available today. For 
instance, air or water jet displays are used to trace outiines on the skin and 
are attractive because they eliminate the need for moving mechanical parts 
25 and because they exhibit a consistent frequency response. Air or water 

bladder displays are used to apply a continuous pressure on an area of skin 
but suffer from extremely low bandwidth. 

Mechanical tactor displays include most of the vibro-tactile devices 
described in the literature in addition to a few displays operated at a very 
30 low frequency that would not norioaLlly be considered vibrotactile. A few 

2 
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Single-element display devices exist but the vast majority of mechanieal 
tactile displays involve .tactor arrays. The number of elements in these 
arrays run from two to 400. Almost all of the devices in this group involve 
> indenting or unpacting die skin perpendicular to the skin surface. Two 

5 niethods of actuation dominate these displays; solenoid activated piiis and 
^ cantilevered piezoelectric bimorph reeds with a pin mounted to the tip. 

Another method of actuation that has great potential is the use of shape 
memory alloys. As a current is passed through the alloy, it changes shape 
and deflects a spring that holds a tactpr pin in place. Voice coil displays 

10 generate vibratory sensations by mounting a voice coil against the skin. 

Electroiactile displays elicit touch sensations by indiscriminately 
stimulating all the touch and pain sensors in the skin with a current piassing 
through the skin. These devices usually involve a remojte stimulator (which 
supplies the power and generates the stimulation waveform) and a number of 

IS electrodes mounted on the skin using a sticky dielectric. Each electrode p^ 
constitutes a stimulation site. While some systenis involve single or a sinall 
number of isolated electrode pairs, arrays of electrodes were also popular, 
with number of elements running up to 1024 in one instance. 

in all of these devices, the sensing or feedback element contains no 

20 intelligence or ability to interact with human stimuli itself. They are hooked 
to a computation engine which contains a world model which interprets the 
raw sensory information and then sends low level commands for behaviors 
to be achievejd. There are cnany limitations in this approach which have 
been apparent and much discussed in the research literature (papers on force 

25 feedback in the presence of time delays by Sheridan and .others) and 
elaborate methods of modeling system performance to overcome these 
problems. The more complex the feedback, the more extreme the problem 
becomes. Ih ^ct this has led the virtual reality rese^ch conmiunity in the 
opposite direction firDm this inyentipn, to use larger and more powerful 

30 computatibnal engines and very simple feedback modes with limited 

3 
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perfonxiance success . Most of the systems used are on non real-time 
computadon engines which makes the problem of providing stable, realistic 
and predictable feedback even more difficxilt. 

The current invention relates to a method and system for overcoming » 

5 these limitations through development of a architecture which provides local 
intelligence and computadon coupled with a method of compressing haptie 
information through various protocols which reduce the amount of 
information that is shared between the world modeler and the haptic 
system. System speed is improved and performance is enhanced making it 

10 feasible to employ multiple modes of haptic feedback simultaneously without 
overloading the communications link or basic computation engines. Since 
multiple .modes of sensory feedback are utilized to reflect the real 
interactions between a human body part and the outside world, a realistic 
user interaction with a. virtual or remote environment is achieved. 

15 Therefore, an object of the invention is to provide local processing to 

each sense and display device so as to provide real-time feedback to the 
human operator. 

Another object of the invention is to provide a modular framework in 
which to extend the sensory feedback capabilities as required by the htmian 
20 operator. 

Another object of the inventioii is to minimize the communications 
bandwidth required between the haptic system and the world modeler. 

Another object of the invention is to provide an anthroponaorphic 
system while retaining portability. 
25 Another object of the invention is to provide a system which can 

sense and/or display force, slip, vibration, shape, stiffness, friction, texture, 
pressure, mass, temperature, sound and .electromagnetic radiation to a 

*. 

human operator. 
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Additional objects and features of the invention will appear from the 
following description in which the preferred embodiments is set forth in 
detail in conjunction with the accompanying drawings. 

i 

m 

5 Summary of the Ijaventipn 

A system for sensing and displaying sensory mfonnation to a human 
operator comprising a world modeler adapted to receive sensory input from 
the operator over a communications medium. The world modeler modifies 
either a virtual w.orld which is a mathematical model of an environment, or 

10 modifies a real world which is a remote real environment. The world 
• modeler then sends sensory information from the real or virtual 
environments oyer the commuhicatioh medium to the haptic system. The 
haptic system displays this sensory infonnation to the human operator. The 
result is that the operator is able to manipulate a real or virtual environment 

15 remotely through the information sensed and displayed by the haptic system. 

The haptic system is composwi of sensory modules. The senisdry 
modules are further composed of a controller (i.e. CPU, DSP or 
microcontroller \init), communications unit, and sensory and/or display 
devices. 

20 The display devices can include heating elements to convey 

temperature, voice coils to coiivey vibration, goggles to display visible, 
motors to display force, brakes to display friction, speakers to convey soimd 
and tactile sensors to convey various senses of touch. 

The sense devices can include encoders to convey position, strain 

25 gages for force, display device feedback to measure torque, thermocouples 
for temperature and cameras for vision. 

Sensory information is compressed dirough various methods. One 
method is to transmit only those sensory cues required to convey the 
intended sense. Another method is to only select the sensory information 



5 



1 



wo 95/20788 PCTAJS9S/01227 

required for a task. Further, the data selected niay be compressed further 
through conventional data compression techniques (i.e. JPEG for video). 

The cpxiLplete system may be scaled as needed. Minimal systems may 
only convey tactile information through a haptic hand > Larger systems .may 
S utilize exoskeletons to convey joint information and force to the system. 

Brief Description of the Drawings 

The invention is described and explained in more detail below using 
the embodiments shown in the drawings. The described and drawn features, 
10 in other embodiments of the invention, can be used individually or in 
preferred combinations. The foregoing and other objects, features and 
advantages of the invention will be apparent from the following more 
particular description of a preferred embodiment of the invention, as 
illustrajted in the accompanying drawings. 
IS FIGURE 1 is a block diagram of the present invention. 

FIGURE 2 is a block diagram of the world modeler of the invention. 

FIGURE 3 is a block diagram of the haptic system of the invention. 

FIGURE 4 is a block diagram of variation of sensory modules of the 
invention. 

20 FIGURE 5 is a block diagram of the invention as implementing 

leleoperation. 

FIGURE 6 is a front view of the hand embodiment of the invention. 
FIGURE 7 is a side view of an example tactile sensory used to 
convey slip. 

25 FIGURE 8 is a block diagram of the haptic system for the hand 

embodiment of the invention. 

FIGURE 9 is a flow chart of the control software for the hand 
embodiment of the invention. 

FIGURE 10 is a front view of the hand, arm and vision embodiment 

w ....... 

30 of the invention. 
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Description of the Preferred Embodiment 
The current inyention relates to a method of overconiing past design 
limitations through development of an architecture which provides local 
intelligence and computation in conjunction with compression of the haptic 
S information through the use of protocols, which reduce the amount of 
information needed to be shared between computation engines at a given 
instant. By utilizing local processing within the sensory units the system 
speed and performance is enhanced making it feasible to employ multiple 
modes of haptic feedback simultaneously without overloading the 
10 communications channel or basic computation engines. Multiple modes of 
feedback are necessary m order to have a realistic interaction between a 

■ 

human body part and a virtual or remote envirooment. 

The present invention generally relates to a system architecture for 
providing sensory feedback in general to the hurnan operator. In particular, 
IS touch feedback is necessary for interactioiis with the human operator in 

controlling either a computer generated, virtual environment or a real remote 

iT 

robotic environment. 

Sensdrv Displav and Sense 

20 The sense of touch, or haptic sense, is complex. Unlike vision, touch 

involves a direct physical interaction, so that human actions cause and 
change what is perceived. It is the fundamental role of touch to sense the 
results of these contact interactions in order to guide manipulation and body 
motion. The current art only contains devices for presenting a crude tactile 

25 image to a location on the body. This type of information represents only a 
fraction of the total information sensed by the human sense of touch. Many 
different types of basic information about the contacts are extracted by 
people through a combination of four different tactile sensory systems, 
measurement of joint motion, and measurement of joint torques . This basic 

30 infonnatipn, can be organized broadly into passively acquired information 
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and actively acquired information. Passively acquired information can be 
gathered without relative motion between the body and an object and without 
varying the forces between the body and the object. Actively acquired 
information requires either relative motion or varying contact forces. A 
suminary of some of the basic types of cues or sensed infonnation is given 
below. 

Passively Acquired Identification Cues 

Passively acquired identification cues can be obtained by the act of 
holding the object in the hand. Three majpr cues have been identi^ed in this 
category: 

1 . Deterniining the local contact normal and curvamre of the 
surface at each contact. 

Det^rminmg the local contact purvature and labeling it as a 
point, edge, or planar contact is important for grasp acquisition and 
object identification. Different contact types have different grasp 
properties which affect stability; Salisbury, J.K. and Mason, M.T., 
Robot Hands and the Mechanics of Manipulation, MIT Press, 
Cambridge, MA (1985). in addition, die local contact normal and 
ctirvature provide a strong pruning heuristic rule for ideiitiiying 
objects and object pose; Grmsonj W.E.L. , Object Recognition by 
Computer: the Role of Geometric Constraint, MfT Press, 
Cambridge, MA (199Q). Local object curvature can be measured 
passively be examining the normal contact force distribution. 

2. Determining the surface texture at each contact. Surface 
texmre affects grasp stability. Rough texmres are generally easier to 
grasp than smooth texmres. In addition, surface texture can also be 
used as a pruning heuristic rule in identifying objects and object pose. 
Texture cues are produced both from the spatial distributidn of the 
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contact force and the dynamic vibration effects produced at the skin 
during motion. 



5 3 . Gross object shape . 

By using the finger geometry and the location of the contact 
points in the hand, the shape of die object can be estimated. For 
example, an initial grasp might be a power grasp in order to get a lot 
of contact information about an object. Once the object and its shape 

10 is identified, ..a dexterous grasp might be used for manipjilatipn. 

Actively Acquired Identification Cues 

With some local active exploratioii, properties relating the reaction 
forces to the applied force can also be determined: 
15 1 . Determining the local stiffness at each contact. 

The local surface sdfftiess can be estimated by applying 
varying normal forces and measuring the change in contact area. 
Softer surfaces will produce a larger contact area for a given applied 
force. This is not a significant cue for manipulation of control 
20 surfaces and tools since almost all of the objects involved are rigid. 

2. Determining local fiictional properdes. 

The local friction along with the local contact type control the 
local grasp stability. Friction can be measured by applying varying 
tangential forces and then detecting the onset of slip (slip detection is 

25 a separate basic manipulatioii cue; see next Section). This task also 

> 

requires the ability to measure tai^ential forces. 

3. Overall object stiffaess. 

•* 

By integrating the contact force over an area with the joint 
torque's, and correlating this with joint deflection, the overall stiffness 
30 can be computed. 
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4. Detecting slip.. 

Detecting the onset of slip between an object and the hand is 
essential for grasp maintenance. Slip detection is used to determine 
the necessary grasp forces at each contact during all states of 
manipulation. 

5. Determining object mass (weight), center of mass, and 
moments of inerda. 

By manipulating a grasped object and measuring the resulting 
joint torque's and net contact forces at the hands for different 
configurations, the weight, center of mass and moments of inertia of 
the object can be computed. This information can be used for object 
identification and pose determination, Siegal, D.M., Pose 
Determination of a Grasped Object Using Limited Sensing^ Ph.D. 
Thesis, Department of Electrical Engineering, MIT, Cambridge, MA 
(1991), as well as for computing the necessary torque's for throwing 
or manipulating the object. 

6. Estimating directions of contact constraint. 

Assenibly is the process or bringing a moving part into a 
constramed relationship with a fixed part. In order to control the 
assembly process, the directions in which movement is constrained 
need to be estimated. Contract constraints are estimated using 
measurements of the reaction forces as a function of position and by 
measuring the direction of impact forces. 

7. Detecting changes in contact constraints . 

This is one of ..the most common tasks during manipulation. 
The detents in switches, the termination in screws, the impacts from 
mating two parts are all examples. The onset of the change can be 

^ — 

detected by looking for unexplained impact forces and the direction of 
the impact force. 
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Current tactile feedback devices present a flat contact image using 
either arrays .of tactors or raster scan techniques. Appendix A includes a list 
of prior art devices whose teachings are herein incorporated by reference. 
This type of display can only be used for very local shape display or 

5 textures, and is Innited to a flat typ^ of display. A heed exists for devices 
for displaying a variety of the remaining cues, $uch as those described 
above, (slip, contact curvature, gross texture) and tiiose cues such as 
vibration, for which devices already exist in a realistic and natural fashion. 
Thus the object of the current invention is to provide a modular structure 

10 under which individual haptic cues or multiple cues can bie effectively and 
efficiently presented to the human operator. 

Haptic Compression 

The bandwidth requured to completely characterize and display haptic 

15 feedback of all senses would normally be prohibitive. Various forms of 
haptic compression are used to limit this bandwidth. 

Haptic compression is based upon human perception. The 
compression algorithms ask the question; what infomiation has to be 
presented to perform this manipulation or recognition task. Haptic 

20 compression is based in sending a set of physical parameters and a number 
indicating the model type to die haptic di$play computer. Locally, the 
model number is used to pull up a stored model of a component of the task 
physics from the memory of the local haptic processor. The parameters then 
control the specific instantation of that model. 

25 For example, perceiving a fine surface texture is done primarily 

through perception of spatial frequencies through vibration. By measuring 
relative velocity and having the spatial frequency stored as a two directions 
fourier spectrum the texmre display would be able to $imulate a complex 
texture. Compression occurs because (1) spatial representation does not 

30 resolve all surface features; (2) vibration is suhstimted for an array display. 

11 




wo 95/20788 PCT/OS95/01227 

Curvature aiid surface normal is based on an analysis of the task 
which shows that only this information is necessary for performing the 
required manipulation task. The conaputer sends 2D curvature properties 
and 1 contact point to represent the surface to the haptic display computer. 
5 The display then takes care of bending up the surface. This is a first order 
representation of the more complex shape. Since not all the information is 
shown» there is information compression. 

For local stiffness, the haptic processor gets a number K which 
represents the contact stiffness. Based on this number it simulates a surface 
10 of stiffness K by applying the correct forces given a displacement relative to 
the display . 

Like local stiffness, a slip haptic processor only needs the coefficient 
of Motion niu. The slip processor can then measure the normal force and 
the tangential forpe to determine the current state of the slip feedback. 
15 In order to in^plement general constant constraints a set of constraint 

4 

equations C_i(g.x) are stored on the local processor. Hie constraint 
parameters g and the specific model type i are sent to the local processor in 
order to instantiate a specific constraint model. The local process applies 
corrective forces to try and force the coordinates of the display, x, to obey 

20 this equation in the display . 

Informatipn may also be compressed, ijhrpugh the removal of 
extraneous information. One method is to include only those senses 
appropriate to the task. A second method is to use psychophysical 
compression, i.e. only including the sense cues necessary to give the 

25 operator the ability to "feel" the characteristic being displayed. A third 
method is to use data compression prior tp transmission across the 
communication channel. A fourth method is data thinning, e.g. 
communicating only the changes since the last transmission or transmitting 
only when changes are large enough to warrant. A fifth method is to 

12 • 
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partition the system such that the computation arid display subsystems share 
a lower bandwidth coramunicaiion channel . 

Sense Selection 

5 There may be no need to include temperamre display, if the task is 

one of simply grasping and moving an object of known temperature. There 
may also be no need to transmit with absolute fidelity all hand jouit torques 
if the application requires only the input of the thumb and forefinger for 
acceptable results. Selection of the.ininimal number of senses required for 

10 convincing haptic feedback is the primary method of reducing 
conununications traffic. 

Psychophvsical Considerations 

Studies have shown that the human may only need a subset of sense 
15 cues to synthesize the sensory feedback. Sense-mapping, or replacing one 
sense for another, can be used for feedback of senses or other data sources 
otherwise difficiilt to display. An example is using a vibrotactile display to 
display surface temperature through variations in vibration amplitude, 

20 Data Compression 

Various methods exist for brute-force data compression. The least 

efficient algorithms, on a percentage compression basis, offer die ability to 

completely and accurately reproduce the input data after decompression. 

More efficient methods, such as fractal compression, reproduce the .data 
25 with reaspnable fidelity and require relatively small transmission bandwidth 

but require some knowledge of the nature of the data being compressed. 

The best protocol will depend on the psychophysics of the senses being 

displayed, 

30 Dara Thinning 

• .13 
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Sending a continuous stream of absolute sensory data across the 
cojcomunicatipns channel may be inefficient. Once the minimum bandwidth 
of the sense displayed is established, only state changes may need .to be 
conmiunicated. The channel could become event driven by transmitting only 

m 

S perceptible change^. 

System Configuration 

The system may he partitioned so that there is a reasonable amount of 
computation performed by the display driving electronics. Each sense can 

10 have it's own processing, sharing and communicating a common knowledge 
of the "world" but reacting in ways unique to the sense being displayed. 
This approach is readily scalable due to its modularity. For example, the 
visual display hardware may be suitied to fast graphics rendering hut does 
not need the ability to compute other sense data. The prdy need is to be 

IS .able to niodify and display its world view given input data from other 
transducers and senses. 

The prototypical haptic niiodule would consist of a processor, 
memory, power and drive electronics, transducer interfacjes, and a 
communications interface and controller. The software on board die haptic 

20 processor would include the kuiematics mapping the sensor data to the 
display. If, for example, finger force feedback was desired, the module 
would query the fmger's position sensors, map this data to joint angles and 
finger geometry in it's world view, sense collisions with objects in it's world 
view, and calculate resultant forces. It would conmiunicate the updated 

25 world view to the other modules and apply the appropriate torques to the 

finger, knowing the configuration of the force feedback device. The system 
is tolerant of system cominunication latencies as the feedback loop is local to 
the haptic processor. 

■ 

30 Scaling and Mapping Sensory Information 

14 
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With the sensors and transducers available, the option of mapping one 
sense to ianother beeomes feasible and perhaps desirable in some cases. The 
ability to map physical parameters not normally sensed (for exiample 
electromagnetic field strengths) to human senses is ootnpelling .and may even 
S lead to more intuitive unders^mding of complicated phenomenon. Sexise 

data naay be freely scaled, such that a remote robot manipulator could 

. . - ^- * 

experience forces on the order of tons that could scale to ounces on the 
hapdc display. At the other end of the spectrum, microfoiices can be scaled 
to pounds at the haptie display, allowing micromanipulation with extremely 
10 fine resolution of control. 

General System Arcfaitecmre 

Referring now to the drawings, wherein like numerals are used 
throughout the several views to identify like elements of the invention, there 

15 is disclosed a system for remote multimode sense and display as shown in 
Fig. l. Hie haptic system 14 interacts with a user 16. Sensory input 19 
from the user 16 is collected by the haptic system 14; This information is 
conapressed by the haptic system 14 and taransmitted onto the 
communications link 12 to the world modeler 10. The world modeler 10 

20 uses the information to manipulate eitiier a real or virtual world. The world 
modeler monitors and measures the state of the world, transmitting display 
information onto communications link 12 to haptic system 14, where the 
display information is converted to sensory output 18 for the user 16. 

With reference to Fig, 2, the world modeler 10 of Fig. 1 may take 

25 two forms 10a or 10b. The first form 10a utilizes a computer system 20 to 
create a mathematical model 22 of a virtual world. The second form of 
world modeler 10b utilizes a computer system 24 to sense information 25 
and send control information 26 to remote input-output system 28. This 
remote input-output system 28 exists in a real environment. 

m 

t 

15 . 
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The haptie system 14 of Fig, 1 is further coinppsed of at least one 
sensory module 30 as shown in Fig. 3. Each Sensory module 30 may Uke a 
variety of formats. Fig. 4 shows examples of sensoiy modules 30a, 30b and 
30c. A sensory module is composed at a controller 4Qa communications 
5 unit 44 and at least one sense unit 42 or display unit 43 . Sensory module 
3Qa contains only one sense unit 42 » sensory module 30b contains only one 
display imit 43, while sensory modide 30c contains two sense rnodules 42 
and one display unit 43. 

Sense unit 42 contains sensors to monitor sensory input from the user. 

.10 For example, this sensory input m^y be force, position, pressure, 

temperature information as experienced by the user. Display unit 43 
contains displays to convey information, such as force, temperature, 
vibration, texture, friction, shape and stiffriess, to the user. The result is the 
user interacts with either a remote or virtual' world through sensory input 

15 controlled by the haptic system. 

Fig. 5 shows the system implementing teleoperation. A user 56 
interacts with the remote world 50. The haptic system 14 gathers sensoiy 
input 52, This information is compressed by the haptic system 14 and 
transmitted on conununications unit 12 to the world modeler 10. The world 

20 modeler generates control informatipn 26 from the compressed sensory input 
received from the communications link 12. The remote input-output unit 28, 
which may be a robot or mechanical arm, responds tp die control 
information 26, thereby acting in the remote envirpimxent 50. The world 
modeler alsp takes sensory input 25 sensed by the remote input-output unit 

25 28 from the remote environment 50. This sensory input 25 is compressed 

by the world modeler 10 and transmitted onto conununications link 12 to the 

haptic system 14. There the compressed sensory information is displayed .54 

to the user 56. The result is that user 56 controls a remote input-output 

device 28 in a remote world 50. 

« 

30 
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Hand Haptic System 

the hand haptic system is comprised of touch sensory modules which 
are composed of a local processing element connected to display and sense 
hardware via wire or fiber optic connection. The sensory module jis in turn 
connected via a wire or fiber optic link or a communications link such as an 
ethernet port, serial poij, bus connector such as a Bit 3, parallel port, game 
port, a local area network to the world model transmitting either direct 
information for simple or low speed applications or information which has 
undergone haptic information compression. The human operator controls 
the world model through interactions with the visual model information 
(displayed on a computer monitor) and the touch display haxdware. 

This results in the jouch display hardware reacting and displaying 
touch information to the operator. The touch sensory module, because it has 
a local processor system, reacts directly to the human stimulus and thus can 
interact at the high speeds required for the feedback to feel realistic even in 
the face of rapidly changing stimuli from the human operator. This reaction 
can be completjely independent from the world model and visual display and 
thus not be slowed by a slow coxnmunications Imk and can therefore rim at 
speeds upwards of IKHz if necessary. This is made possible because of the 
architecture of the local processor unit. This processor can be a singje chip 
such as a DSP or Microcontroller, a board witii rnidtiple chips or a 
computer, depending upon the number of sensor and display elements 
contained in the touch display hardware and the complexity of the model. 

The haptic compression is achieved by utilizing the basic research mtp 
human perception of touch described above to determine which touch cues 
are being utilized for a given type of manipulation. For example, when m 
object is grasped, the joint torcjue cues as well as slip .information are used to 
ensure that the object is stably grasped. The algorithms in the local 
processor determine the type of cues required and their magmmde thus 
reducing the amount of information which needs to he transmitted to the 

17 
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Other system components. The world model only needs to know the 
resulting object status (i.e. position and orientation, or forces acting on the 
object, depending upon how the world model is structured) thus the 
information content of the transmitted signals is greatly reduced thus 
5 improving the ovierall perforinance of the system. 

In the embodiment shown in Fig. 6, three sense and display elements 
were used; slip on the finger tips, force feedback to the index finger and 
thumb, and vertical resistance. In addition the device provides position and 
orientation of each of the system elements. 

10 . In this embodiment a simple slip display, shown in Fig. 7, was 

developed as disclosed in U.S.S.N. 08/187,646, entitled "Multimode 
Feedback Display Technology " filed January 27, 1994 (herein incorporated 
by reference) . The slip display consists of a delrin cylinder 71 with a 
textured surface, durectly mounted o» the shaft of a miniature gear motor 72. 

15 The cylinder is supported on the other side with a miniature bearing 73, 
The motor is held by a set screw in the mounting bracket 70. 

In Fig. 6, the slip display 61 was attached to the tip of each of the 
index finger force fe$^back apparatus 62 and the thumb force feedback 
apparatus 63. Force reflection to the index finger is provided by a modified 

20 two-degree-of-fireedom SAHRE (Sensing and Force reflecting Exoskeleton^ 
Patent application serial #07/961,259 submitted 10/15/92) prototype, which 
consists of two cable-driven linkages 65, with the two motors 66 mounted on . 
the forearm. The axes of rotation of the linkages are co-located with the 
PIP and MCP joints of the * index finger. Force refleciion to the thumb is 

25 provided by a oneHdegree-of-freedom device 67, also driven by cables 69 
from a remote motor mounted on the forearm 68. 

The vertical resistance and position and orientation in space is 
provided by a boom 64. The vertical force reflection axis provided up to 2 
lbs of virtual weight at the hand. It also supported the weight of the 

30 S AFIRE throujgh counterbalancing. The stiffness of the vertical boom up to 
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the S AFIRE attachment bracket exceedeid 140 Ib/in, the miniinum stiffness 
required for a cantilevered beam to feel rigid to a human operator. The 
resolution of the vertical and .horizontal position sensors for the hand and 
index finger was 0.005 inch. Two linear ways were mounted on opposite . 
sides of the vertical support structure, which was rigidly attached to a heavy 
base. 

The drive mechanism 201 for the vertical degree-of-freedom was 
located on the back of the support structure 200. The motor providing 
vertical force reflection was borizontsaiy mounted oh the slider riding in the 
rear linear way. A threaded capstan was directly mounted on the shaft of 
the motor. A 7x7, 0.032 mch diameter steel cable was wound 3-4 turns 
around the capstan. One end of the cable was terminated oh a cable 
tensioner at the top of the structure. As the motor rotated, the cable hoisted 
the slider up and down the rear linear way. 

The rear slider was connected through a second cable aiid a set of 
pulleys to a plate mounted on two sliders riding in a second linear way, 
mounted on the front side of the vertical support structure. Using two 
sliders instead of oue greatly improved the twisting and bending load 
capacity of the front linear way . The two sliders were mounted on the plate 
through two pivots to minimize bending moments applied to the front linear 
way. 

Vertical forces were transmitted to the SAFIRE by a horizontally 
mounted linkage. The linkage consisted of a distal block attached to the 
front slider plate, proximal block, and a 12 inch long rod. The distal block 
housed the bearings and the encoder for the first passive pivot, also called 
the distal pivot. One end of die rod was pivoted about this block so that thfe 
luakage rotated freely in the horizontal plane. The other end of the rod was 
pivoted about a second bearing/encoder housing, also called the proximal 
pivot. The proxunal mounting block was attached to the SAFIiRE base with 
a rigid bracket. The length of this bracket was designed to position the hand 
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SO that the proximal pivot was between the tip of the index finger and the 
thumb. 

The motor for vertical force reflection was selected based on the 
power requirements. Ideally the motor should produce a force of 2 lbs at 

5 the hand . It is estimated the hand could move at a ina;cunum vertical speed 
of 8 in per sec. We selected a 90W motor without a gearhead to provide a 
smooth operation with minimal friction. This is designed to produce a force 
that exceeds the corresponding frictional force, if the frictional coefficient 
between the finger and the display is estinaated at 0.5, the tactile display 

10 must be able to produce 1 lb. of force. If the display moved past the finger 
at 1 in/s, the required power would be around 0.112W. A motor producing 
0. 25-0.5 W would therefore be a suitable actuator for this display. The 
MicroMo Series 1016 motor fits these requirements; it is 0384 inch in 
disuneter, 0.63 inch long and weighs 0-23 oz. This is mounted in the space 

15 beyond the fingiertip. 

The hand haptic system is interfaced to the world modeler, in this 
case implemented on a graphics rendering PC, through a local processor .80, 
in this case a VME card cage which houses a TI C30 DSP board 81, I/O 
boards 82, 83, and 84, and a BiT3 VME-ISA bus adapter 85. 
.20 The physical sirciulation and device drivers reside on a TI C30 

floating point DSP board :81. The DSP board is installed in a multislpt 

VME cage which also houseis the I/O electronics. The simulation is 

p ' ...... 

interrupt driven with a lime step of 300 Hz, which is the minimum 
performance level necessary for a stable and realistic simulation. Higher 

25 performance could be gained by parsing out the simulation tasks among 
sfeveral DSP processors and executing them in parallel. For example, 
communication tasks with the graphics PC and boom position sensing could 
be handled by a different processor than the main simulation loop. 

The DSP simulation, as shown in Fig. 9, communicates with the 

30 graphics RC via a BiT3 VME-ISA bus adapter. This high speed 

20 
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CQitununicadons channel handles graphics infpnnatioii generated by the 
simulatipB and simulation parameters generated by the user interface. The 
user interface starts the standard simulation demonstration and provides 
program stams informatioii. Since the user interfece and graphics rendering 

5 can be somewhat asyiichronous to the simulation, graphics and PC system 
latencies will not dismrb the simulation which depends on a uniform time 
step. Also the bandwidth xequired for this information is Ipw enough to not 
require a dedicated processor. This ^software approach does not require the 
simulation and graphics to communicate a significant amount of objects 

10 characteristics upon starmp of the simulation so that a minimum of data is 
exchanged during run time. Data from the simulation need not be 
transmitted to the graphics PC any faster than the graphics refresh rate. For 
example, at a 60 Hz refresh rate, new data would be needed at a m a x imu m 
of every third integration time step. 

15 Sensing the joint.positions and orientations and applying appropriate 

forces and slip requires additional I/O electronics. The joint positions are 
sensed with digital position encoders. The joint torque's and slip applied to 
the touch display are voltage controlled by the simulation via 6 12-bit digital 
to analog converters and external current mode motor amplifiers. The 

20 encoder coimters and digital to analog converters reside on the high-speed 
VME bus witihi the DSP board and are directiy accessed by the DSP. The 
• generality of the control electronics and overall architecture makes die 
system adaptable to other force reflecting masters. The minimum 
configuration can handle six output and eight input degrees of freedom, and 

25 can be easily expanded to any reasonable number by adding I/O modules 
iand, if necessary , DSP processors. 

Almost all of the application graphics and DSP code is written in C. 
The simulation can be developed on the graphics PC and debugged with 
tools available for the PC/BiT3 development environment. Porting the 

30 , simulation to the DSP is then relatively straightforward. In fact, the existing 
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simulation can be debugged on the PC and dien recompiled for the DSP with 

m 

only minor changes. Hardware based debugging tools are available for the 
DSP, although the DSP/PC interface provides a relatively powerful 
Stethoscope for iall but the most low level debugging or obscure 
S compiler^iardware bugs. The DSP executable and source code resides on 
the PC and is downloaded to the DSP at run time. . 

This system and method can be applied to a wide variety of physical 
embodiments including simpler systems and more complex and capable 
systems with more feedback modes utilized, 

10 

Hand/Arm/Vision Hantic Svstem 

An ailtemate embodiment utilizes the hand haptic system in 
conjunction with an exoskeletal arm mechanism combined with a vision 
display. The exoskeletbn arm consists of two uxterconnected mechanisms, 
IS one for the shoulder and the other for the elbpw and forearm as shown in 
Figure 10. 

The shoulder mechanism 90 consists of a three DQF motor driven 
gimbal, a passive prismatic joint (spline), a 2 DOF passive gimbal and a 
upper arm cuff. The elbow mechanism 92 is connected to the shoulder 

20 mechanism 90 at the distal end of the spline 91. This allows the system to 
acconunodate changes in the distance between the shoulder 90 and elbow 
mechanisms 92. The spline also elirninates the need for adjustment for 
different upper arm lengths. 

The center of the distal gimbal 93 is located on the axis of the upper 

25 arm E/I rotation. The elbow F/E motor 94 is pivotally jointed to both the 
upper 95 and lower 96 cuffs. The final step was to use a brake 97 at the 
distal gimbal of the shoulder mechanism to ground the elbow/forearm 
mechanism. 



22 



95/20788 



PCtAJS95/01227 



the elbow mechanism has two DOF, The first DGF is aligned with 
the axis of elbow F/E. The second DOF is placed along the forearm with a 
remote center drive mechanism. 

The three motors 100, 101, and 102. at the shoulder gimbal can he 
CGordmated to apply a force .at the upper arm which is approxiinately 
perpendicular to and intersects with the longimdinal axis of the upper aim. 
This force creates a torque about axes of the shoxilder F/E and Ab/Ad. The 
motor at the elbow controls the Flexion/Extension torque about the elbow. 
The rotational torque about the longitudinal axes of the upper and lower 
arms are controlled in two different ways depending qn the position of the 
elbow joint. When the elbow is extended the motor at the wrist applies 
rotational torque to the longitudinal aces of both the upper and lower arms, 
since their motions coincide. When the elbow is flexed, the motors at the 
shoulder can be poordinated to apply a force at the forearm cuff to produce 
a torque about die upper arm E/I, and the RCD applies a torque to the 
forearm. 

The biggest advantage of this design was that it conformed to the 
anatomic modons of the eiitire arm. It imposed no kinematic constraints on 
the arm because the shoulder mechanism had fiiU 6 DOF and the elbow F/E 
mechanism formed a 4-bar linlcage with the elbow F/E joint. Therefore, it 
was very comfortable to wear. It also offered large Range Of Motions 
(ROM) . It not only applies forces to the entire arm grounded to the back, 
but also torque to individual joints. The former allowed us to simulate 
picking up an object by bracing between the forearm and the upper arm or 
between the entire arm and the chest. The latter could simulate contacts of 
objects in a virtual envixpiflneiit by any portion of the arm such as hand, 
foreann and upper arm. In summary, this mechanism offered all the force 
display requirements for a force reflective arm master and corifonned to the 
anatomic motion of the aim. 
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A higher performance version can be achieved by using magnetic 
brakes in conjunction to tow miniature motors or actuators. The two 
magnetic particle brakes may be used to replace the f^-st two motors at the 
shoulder gimbaL In order to use brake3, the mechanism must be fuUy 
S balanced about these two axes because brakes are passive and cannot lift. In 
addition, the niiniature actuators or motors must be put in series with, and 
parallel to, the two brakes for active force control. One way to achieve this 
is to activate the 2 DOF passive giinbal at the upper arm cuff with such 
miniature motors. With the brakes locked, die two miniature actuators or 
10 motors can display small forces to ^e operatpr at very fme resolutions. To 
simulate a hard contact, simply lock the brakes and drive the two miniature 
motors to their respective hard stops so that the braking forces can be felt by 
the upper arm. For a force in between these two extremes, the combination 
of Pulse Width Modulations (PWM) on the brakes and force controls oh the 
15 miniature motors can be used simultaneously. 

The video sense is accommodated via a h^d mounted display or 

• * 

remote video screen, with accompanying processing. They can 
communicate with each haptic processor, which take in data from the 
physical parameters of the sense being displayed, e.g. position of the fingers 

20 related to force feedback. Each haptic processor applies the appropriate 
feedback, and communicates its part of the world view to the other 
processors. The communication chaimels can be fibre optic, infrared, 
wired, or whatever medium is appropriate to the twin demands of bandwidth 
and portability. Adding another sense is as straightforward as adding 

25 another sense module up to the limitations of inter-module bandwidth. 
System autocpnfiguration is a straightforward function of the softwar€| 
Controllers for the modules could be general puxpose digital signal 
processors for maximum flexibility or specialized processors, e.gi 
application specific integrated circuits, for minhnum cost. 
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Vision sense informajdon is provided to the operator through a set of 
goggles containing vision displays . Such a device is the Vision Immersion 
Module manufactured by Kaiser Electro-Optics in Carlsbad, CA 92008. 
Sound may also he provided to the operator through a set of headsets. 

5 

Portable System 

■ • 

The complete system is worn by the human operator through a 
backpack mechanism which also contains the electronic and power systems. 
The resulting system is a self-supporting exoskeleton, requiring no precise 
10 alignment to conform to the anatomical movement of the arm, and is thus 
comfortable to wear . The device is comfortable to use because it applies 
normal forces to the arm; no twist, no shear to skin or bones. It is also easy 
to don and doff because entire mechanism is supported through the 
backpack. 

IS The complete system communicates to the world modeler through a 

cable which also provides power to the system. The system can ako be self 
contained by providing power through batteries and conununications through 
an RF or IR link. 

The techniques of haptic compression, combined with modem power 

20 electronics and digital signal processing, rnake it possible to construct a 

totally portable haptic display systena. . Such a system can be used to explore 
virtual worlds in architecmral applications, control remote mobile robots in. 
space or nuclear applications, or transverse any number or databases to 
explore them through haptic feedback " 

■ 

25 While the invention has been particularly shown and described widi 

reference to the preferred embodiments thereof ^ it will be understood by 
those skilled in the art that varipus changes m form and detail may he nptade 
therein without departing from the spirit and scope of the invention. 
What is claimed is: 

■ 
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1 . A system for sensing and displaying sensory information to a user 
comprising: 

an world modeler adapted to receive first sensory information and 
transmit second sensory information, 
5 a communications mediuni adapted to carry the first sensory 

infonnation and the second sensory information, 

a haptic system adapted to sense the first sensoiy informatipn and 
transmit the first sensory information on the communications medium to the 
environment creator, adapted to receive the second sensory information 
10 transmitted on the communications medium by the environment creator and 
further adapted to display the second sensory infonnation, 

whereby the system senses sensory information from the tiser and 
displays sensory infonnation to the user. 

15 2. The sysjem of claiin r wherein the world modeler is a computer 

system, 

the computer system adapted to create a matheniatical model of an 
environment, 

the computer system further adapted to manipulate the matbenoatical 
20 model of an environment based upon the received first sensory information, 

the computer system further adapted to create the second sensory 
information based upon the mathematical model of an environmient and the 
first sensory information, 

whereby the poniiputer system creates an artificial environment for the 
.25 user to interact. 



3 . The system of claim 1 wherein the world modeler is a conaputer 
system and a reniote input-output device, 

liie remote input device adapted to sense third sensory information 
30 from the remote ihput-ouiput device, 
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the computer system adapted to manipulate the third sensory 
information to create the second sensory information, 

the coniputer system further adapted manipulate the first sensory 
informatidn to create fpunh sensory information, 

* 

5 the remote input-output adapted to respond to the fourth sensory 

information, 

whereby the remote input-output device is interactively controlled 
through the computer systein by the user. 

10 4. The system of claim 1, 2 or 3 wherein the haptic systena 

comprises: 

at least one sensory module adapted to: 
sense direct sensory information, 

process the direct sensory iirforination iiito the first sensory 

IS information, and 

transmit the first sensory information, 
whereby the sensory module communicates the direct sensory 
ixiformation processed by the sensory module to die world modeler. 

20 5. The systena of claim 4 wherein die sensory module is further 

adapts to : 

. receive the second sensory information, 
process the second sensory information into direct display 
information, and 
25 display the direct display information, 

whereby the sensory module displays the direct display infomiation 
processed by the sensory module to the user. 
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Fig. 1 



18- 



Environment 
Creator . 



Contmunications Link 
Sense and Olsptay lnfonnatiori 



0 



Haptic 
System 



Sense 



Display 



12 



10 



19 



20 



1/7 



wo 95/20788 



PCT/US95/01227 



Fig. 2 
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FIG. 3 
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